Abstract
Selecting the best hyperparameter configuration is crucial for the performance of machine learning models over large-scale data. To this end, the automation of hyperparameter optimization (HPO) has been widely applied in many automated machine learning (AutoML) frameworks. However, without the effective mechanisms of early stopping and prior knowledge leveraging, such automation is often time-consuming and even inefficient. To improve efficiency, we introduce AntTune, a distributed HPO system that includes parallel optimization, distributed evaluation, tensor cache, etc. Specifically, in AntTune, a time-saving and lightweight mechanism of early stopping is designed to process multiple trials simultaneously. Also, a tree-based meta-learning approach is developed to leverage knowledge from prior tasks and thus it can speed up current HPO tasks. The extensive experiments on both public and industrial datasets demonstrate that our AntTune can improve the state-of-the-art HPO platforms by an average of 3.26% in terms of the effectiveness metrics and 26.25% in terms of tuning time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
UCI URL: https://archive.ics.uci.edu/ml/index.php.
References
Bergstra, J., Yamins, D., Cox, D.: Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In: ICML, pp. 115–123. PMLR (2013)
Chen, T., Guestrin, C.: Xgboost: a scalable tree boosting system. In: SIGKDD, pp. 785–794 (2016)
Cheng, H.T., Koc, L., Harmsen, J., et al.: Wide & deep learning for recommender systems. In: DLRS, pp. 7–10 (2016)
Erickson, N., Mueller, J., Shirkov, A., et al.: Autogluon-tabular: robust and accurate automl for structured data. arXiv preprint arXiv:2003.06505 (2020)
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J.T., Blum, M., Hutter, F.: Auto-sklearn: efficient and robust automated machine learning. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning. TSSCML, pp. 113–134. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05318-5_6
Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25566-3_40
Jin, H., Song, Q., Hu, X.: Auto-keras: an efficient neural architecture search system. In: SIGK, pp. 1946–1956 (2019)
Kandasamy, K., Vysyaraju, K.R., Neiswanger, W., et al.: Tuning hyperparameters without grad students: scalable and robust bayesian optimisation with dragonfly. arXiv preprint arXiv:1903.06694 (2019)
Le, T.T., Fu, W., Moore, J.H.: Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics 36(1), 250–256 (2020)
Li, L., Jamieson, K., Rostamizadeh, A., et al.: A system for massively parallel hyperparameter tuning. MLSys 2, 230–246 (2020)
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., et al.: Hyperband: a novel bandit-based approach to hyperparameter optimization. JMLR 18(1), 6765–6816 (2017)
Microsoft: Neural Network Intelligence. https://github.com/microsoft/nni, Accessed 30 July 2022
Salesforce: Transmogrifai’s documentation. https://transmogrif.ai/. Accessed 8 July 2022
Thornton, C., Hutter, F., Hoos, H.H., et al.: Auto-weka: combined selection and hyperparameter optimization of classification algorithms. In: SIGKDD, pp. 847–855 (2013)
Yu, Y., Qian, H., Hu, Y.Q.: Derivative-free optimization via classification. In: AAAI (2016)
Zhou, J., Velichkevich, A., Prosvirov, K., Garg, A., Oshima, Y., Dutta, D.: Katib: a distributed general automl platform on kubernetes. In: OpML, pp. 55–57 (2019)
Zimmer, L., Lindauer, M., Hutter, F.: Auto-pytorch: multi-fidelity metalearning for efficient and robust autodl. IEEE TPAMI 43(9), 3079–3090 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, J., Shi, Q., Ding, Y., Wang, L., Li, L., Zhu, F. (2023). AntTune: An Efficient Distributed Hyperparameter Optimization System for Large-Scale Data. In: Wang, X., et al. Database Systems for Advanced Applications. DASFAA 2023. Lecture Notes in Computer Science, vol 13946. Springer, Cham. https://doi.org/10.1007/978-3-031-30678-5_35
Download citation
DOI: https://doi.org/10.1007/978-3-031-30678-5_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30677-8
Online ISBN: 978-3-031-30678-5
eBook Packages: Computer ScienceComputer Science (R0)